1. Field of the Invention
This invention is related to the field of computer systems and, more particularly, to increasing bandwidth for issuing ordered transaction into a distributed processing system.
2. Background of the Related Art
Generally, personal computers (PCs) and other types of computer systems have been designed around a shared bus system for accessing memory. One or more processors and one or more input/output (I/O) devices are coupled to memory through the shared bus. The I/O devices may be coupled to the shared bus through an I/O bridge which manages the transfer of information between the shared bus and the I/O devices, while processors are typically coupled directly to the shared bus or are coupled through a cache hierarchy to the shared bus.
Unfortunately, shared bus systems suffer from several drawbacks. For example, the multiple devices attached to the shared bus present a relatively large electrical capacitance to devices driving signals on the bus. In addition, the multiple attach points on the shared bus produce signal reflections at high signal frequencies which reduce signal integrity. As a result, signal frequencies on the bus are generally kept relatively low in order to maintain signal integrity at an acceptable level. The relatively low signal frequencies reduce signal bandwidth, limiting the performance of devices attached to the bus.
Lack of scalability to larger numbers of devices is another disadvantage of shared bus systems. The available bandwidth of a shared bus is substantially fixed (and may decrease if adding additional devices causes a reduction in signal frequencies upon the bus). Once the bandwidth requirements of the devices attached to the bus (either directly or indirectly) exceeds the available bandwidth of the bus, devices will frequently be stalled when attempting access to the bus, and overall performance of the computer system including the shared bus will most likely be reduced.
On the other hand, distributed communication systems lack many of the above disadvantages. A distributed communication system includes multiple nodes interconnected by multiple independent communication links. The distributed communication system thus may include many different paths via which communication traffic among the nodes may travel, and traffic may pass through intermediate nodes enroute to a final destination. The distributed communication system also may include a distributed memory having multiple portions, each of which is located at a particular node. A memory address space of the computer system is assigned across the memories portions at each node.
In general, a “node” is a device which is capable of participating in transactions upon the interconnect. For example, the interconnect may be packet based, and the node may be configured to receive and transmit packets as part of a transaction. Generally speaking, a transaction is a series of packets. A “requester” or “source” node initiates a transaction directed to a “target” node by issuing a request packet. Each packet which is part of the transaction is communicated between two nodes, with the receiving node being designated as the “destination” of the individual packet. When a packet ultimately reaches the target node, the target node accepts the information conveyed by the packet and processes the information internally. Alternatively, a node located on a communication path between the requester and target nodes may relay the packet from the requester node to the target node.
In addition to the original request packet, the transaction may result in the issuance of other types of packets, such as responses, probes, and broadcasts, each of which is directed to a particular destination. For example, upon receipt of the original request packet, the target node may issue broadcast or probe packets to other nodes in the processing system. These nodes, in turn, may generate responses, which may be directed to either the target node or the requester node. If directed to the target node, the target node may respond by issuing a response back to the requester node.
Distributed communication systems present design challenges which differ from the challenges in shared bus systems. For example, shared bus systems regulate the initiation of transactions through bus arbitration. Accordingly, a fair arbitration algorithm allows each bus participant the opportunity to initiate transactions. The order of transactions on the bus may represent the order that transactions are performed (e.g. for coherency purposes). On the other hand, in distributed communication systems, nodes may initiate transactions concurrently and use the interconnect to transmit the transactions to other nodes. These transactions may have logical conflicts between them (e.g. coherency conflicts for transactions involving the same address) and may experience resource conflicts (e.g. buffer space may not be available in various nodes) because no central mechanism for regulating the initiation of transactions is provided. Accordingly, it is more difficult to ensure that information continues to propagate among the nodes smoothly and that deadlock situations (in which no transactions are completed due to conflicts between the transactions) are avoided.
Another challenge presented by a distributed communication system involves control of the ordering of transactions directed to memory. Because transactions directed to a target may travel different paths, there is no assurance that a particular transaction will reach the target prior to a subsequently issued transaction. Uncertainty or ambiguity with respect to transaction ordering may be problematic in certain systems in which memory requests (e.g., read and write transactions) may need to be properly ordered with respect to other pending memory operations to preserve memory coherency within the computer system and/or to satisfy ordering requirements expected by I/O subsystems (e.g., PCI). For example, memory operations may need to be completed in the order in which they were generated. It would thus be desirable to provide a computer system implementing a system and method to control and ensure proper ordering of transactions in a distributed communication system.
Maintaining ordering of transactions can create lengthy delays between the time a first transaction is issued and the time a second transaction may be safely issued without disturbing ordering. For example, to ensure that ordering is maintained with respect to two transactions, the second transaction may be stalled until a previously issued first transaction has completed. It would thus be desirable to increase the bandwidth for ordered transactions by providing a system and method which delays issuance of the second transaction for only as long as needed to ensure that the second transaction will be properly ordered with respect to the second transaction.
The present invention may be directed to one or more of the problems set forth above.